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Preface 



This is one of three documents constituting the mandated Final Report on the National Assessment of 
Tide I. This Summary of Key Findings contains findings from both Volume I, Implementation, and Volume 
II, Closing the Reading Gap: Findings from a Ruindomie^ed Trial of Four Reading Interventions for Striving Keaders. 
Volume I was prepared by the U.S. Department of Education, Office of Planning, Evaluation and Policy 
Development, Policy and Program Studies Service. Volume II was prepared by the Corporation for the 
Advancement of Policy Evaluation. The National Assessment of Title I Interim Report was released in 
April 2006. 
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I. Introduction 



The No Child Left Behind Act of 2001 (NCLB), which first went into effect beginning with the 2002-03 
school year, reauthorized the Tide I program and made a number of significant changes. NCLB 
strengthened the accountability provisions of the law, requiring that states establish assessments in each 
grade from 3-8 and once in grades 10-12, and set annual targets for school and district performance that 
would lead to all students reaching proficiency on those assessments by the 2013-14 school year. 

Schools and districts that do not make adequate yearly progress (AYP) towards this goal are identified as 
needing improvement and are subject to increasing levels of interventions designed to improve their 
performance and provide additional options to their students. NCLB also required that all teachers of 
core academic subjects become highly qualified, which the law defines as holding a bachelor’s degree and 
full state certification, as well as demonstrating competence, as defined by the state, in each core 
academic subject that he or she teaches. These and other changes were intended to increase the quality 
and effectiveness not only of the Title I program but of the entire elementary and secondary education 
system in raising the achievement of all students, particularly those with the lowest achievement levels. 

This report expands and updates the information provided in the National Assessment of Title I Interim 
Keport that was released in April 2006. 

• New data on the implementation of Title I are provided on the targeting and uses of Title I 
funds, services for private school students, characteristics of students participating in the school 
choice and supplemental services options, achievement trends on the NAEP science assessment, 
and surveys of parents and supplemental service providers. Updated data on student 
achievement on state assessments, school and district identification for improvement, and highly 
qualified teachers, and schools’ AYP and improvement status are also provided. 

• New data are provided for third- and fifth-graders who participated in one of four promising 
remedial reading interventions during the 2003-04 school year as part of the Closing the Teaching 
Gap study. These students’ reading skills were assessed again in spring 2005, one year after the 
end of the intervention, to evaluate sustained effects of the interventions. 



A. National Assessment of Title I 

As part of NCLB, the Congress mandated a National Assessment of Title I (Section 1501) to evaluate 
the implementation and impact of the program. This mandate required the establishment of an 
Independent Review Panel (IRP) to advise the Secretary on methodological and other issues that arise in 
carrying out the National Assessment and the studies that contribute to this assessment. In addition, the 
law specifically requires a longitudinal study of Title I schools to examine the implementation and impact 
of the Title I program. 



1 





On November 6, 2002, the President signed the “Education Sciences Reform Act of 2002,” establishing 
a new National Center for Education Evaluation and Regional Assistance (NCEE) in the Institute of 
Education Sciences. Part D of this Act assigned responsibility for the National Assessment of Tide I to 
NCEE. The creation of this Center represented an important shift in the purposes of program 
evaluation and the types of methodology used in Department evaluation studies from broader policy and 
program assessments to specific scientific evaluations of program effectiveness. In the past. Department 
program evaluation studies of Tide I have focused primarily on issues of program implementadon, such 
as targedng of federal resources, compliance with federal laws and reguladons, characterisdcs of program 
pardcipants, and types of services provided. 

However, school superintendents, principals, and teachers often do not have the informadon they need 
in order to make sound decisions to improve instrucdon and raise student achievement. In many areas, 
the sciendfic evidence on the effecdveness of educadon programs is weak, inconsistent, or nonexistent. 
Evidence is needed on the effecdveness of specific intervendons to inform Tide I program 
improvement. NCLB repeatedly emphasi 2 es the importance of adopdng sciendficaUy proven 
educadonal pracdces and programs. In an effort to significandy raise the quality of sciendfic evidence on 
program effecdveness, NCEE has launched a generadon of evaluadon studies that use the most rigorous 
evaluadon designs possible to detect the impact of educadonal pracdces and programs on student 
achievement. Under the Nadonal Assessment of Tide I, NCEE has begun studies of remedial reading 
programs, reading comprehension programs, and mathemadcs curricula to assess the effecdveness of 
educadonal programs in these important areas of academic achievement. These studies are randomi 2 ed 
trials in which schools or teachers are randomly assigned to an educadonal program or to the control 
condition. Such experimental designs are the most reliable and accurate way of esdmadng the 
effecdveness of an educadonal intervendon. 

In response to the requirements in Secdon 1501, the Nadonal Assessment of Tide I includes both 
studies of program implementadon and of the effecdveness of specific intervendons. Implementadon 
studies, carried out by the Policy and Program Studies Service (PPSS) in the Office of Planning, 
Evaluadon and Policy Development, provide nadonaUy representadve data on the implementadon of key 
components of the Tide I program. Effecdveness studies, carried out by the Institute of Educadon 
Sciences, provide evidence about those pracdces which produce the best results. Together these two 
types of studies can provide the informadon needed to effecdvely target technical assistance and assist 
policymakers in making decisions on the best use of resources. 



B. Independent Review Panel for the National Assessment of Title I 

The mandated funcdon of the Independent Review Panel (IRP) for the Nadonal Assessment of Tide I is 
to advise on methodological and other issues that arise in carrying out the assessment. The IRP is to 
ensure that the assessment and studies adhere to the highest possible standards of quality with respect to 
research design, stadsdcal analysis, and the disseminadon of findings; and that the studies use valid and 
reliable measures to document program implementadon and impacts. The IRP was appointed in 
November 2002 and is made up of researchers, educadon pracddoners, parents, and members of other 
organi 2 adons involved with the implementadon and operadon of programs under Tide I. A list of 
current IRP members and their affikadons is included in Appendix A. 

The IRP first met in January 2003 and has been instrumental in shaping the direcdon of implementadon 
and effecdveness studies under the Nadonal Assessment of Tide I. At this meedng, the IRP noted that 
an evaluation of the impact of Tide I funds on student achievement was not feasible because it would 
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require random assignment of Title I funds to eligible districts and schools. Past evaluations of activities 
supported by Tide I have provided Uttie information on how to improve student achievement. The IRP 
recommended that Tide I effecdveness studies focus on “what works” evaluadons of well-defined 
intervendons for improving achievement of high-poverty students in the cridcal areas of reading and 
mathemadcs. These evaluadons would provide informadon on the effecdveness of specific 
intervendons that could be adopted by schools to improve academic achievement. Addidonal 
informadon on IRP recommendadons for effecdveness studies is included under “Future Reports” in 
Chapter III of this document. 

The IRP has also provided essendal advice on the conduct of implementadon studies. At its first 
meedng, the panel agreed that the mandated nadonal longitudinal study of Tide I schools should be 
launched as soon as possible, and most members advised that it should focus on program 
implementadon rather than the impact of federal funds for reasons described above. The IRP 
recommended that the Nadonal Longitudinal Study of No Child Left Behind include a survey of parents 
concerning the Tide I school choice and supplemental educadonal services opdons in the law. 

The IRP has met eight dmes over the past three years. Several meetings were held in the first year after 
the panel’s appointment in November 2002: January 30-31, 2003; March 17-18, 2003; September 22, 
2003; and November 9, 2003. There were also IRP meedngs on November 21, 2004, on July 29, 2005, 
and on June 14-15 and October 15-16, 2006. The IRP has provided valuable advice on the design and 
implementadon of the Tide I studies as well as extensive comments on reports from the Nadonal 
Assessment. 

C. Key Provisions of Title I under the No Child Left Behind Act 

NCLB, which went into effect beginning with the 2002-03 school year, strengthened the assessment and 
accountability provisions of the law, requiring that states annually test all students in grades 3-8 and once 
in grades 10-12 with assessments that are aligned with challenging state standards. States must also set 
targets for school and district performance that lead to all students achieving proficiency on state reading 
and mathemadcs assessments by the 2013-14 school year. Schools and districts that do not make 
adequate yearly progress (AYP) towards this goal for two consecudve years are idendfied as needing 
improvement and are subject to increasing levels of intervendons designed to improve their 
performance, as well as to provide addidonal opdons to their students. In schools idendfied for 
improvement, districts must offer students the opdon to transfer to another school. If an idendfied 
school misses AYP again (for a third year), low-income students in the school must be offered the 
opdon to receive supplemental educadonal services from a state-approved provider. If an idendfied 
school misses AYP for a fourth year, the district must take one of a set of “correcdve acdons” specified 
in the law, and if the school misses AYP for a fifth year, the district must begin planning to restructure 
the school. 

NCLB also requires that all teachers of core academic subjects become “highly qualified,” which the law 
defines as having a bachelor’s degree and full state cerdficadon as well as demonstradng competence, as 
defined by the state, in each core academic subject that he or she teaches. Exhibit 1 provides a more 
detailed summary of key NCLB provisions. 
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Exhibit 1 

Key Provisions of the No Child Left Behind Act 


State 

assessments 


States must implement annual state assessments in reading and mathematics in grades 3-8 and at least 
once in grades 10-12, and in science at least once in each of three grade spans: 3-5, 6-9, and 10-12. 
Assessments must be aligned with challenging state content and academic achievement standards. 
States must provide for participation of aU students, including students with disabilities and limited 
English proficient (LEP) students. States must provide for the assessment of English language 
proficiency of aU LEP students. 


Adequate 
yearly progress 
(AYP) 


States must set annual targets that will lead to the goal of aU students’ reaching proficiency in reading 
and mathematics by 2013-14. For each measure of school performance, states must include absolute 
targets that must be met by key subgroups of students (major racial/ ethnic groups, low-income 
students, students with disabilifies, and LEP students). To make AYP, schools and districts must 
meet annual targets for each student subgroup in the school, and must test 95 percent of students in 
each subgroup. States also must define an “other academic indicator” that schools must meet in 
addition to proficiency targets on state assessments. 


Schools 
identified for 
improvement 


Title I schools and districts that do not make AYP for two consecutive years are identified for 
improvement and are to receive technical assistance to help them improve. Those that miss AYP for 
additional years are identified for successive stages of interventions, including corrective action and 
restructuring (see below). To leave identified-for-improvement status, a school or district must make 
AYP for two consecutive years. 


Public school 
choice 


Districts must offer aU students in identified Title I schools the option to transfer to a non-idenfified 
school, with transportation provided by the district. 


Supplemental 

educational 

services 


In Title I schools that miss AYP for a third year, districts also must offer low-income students the 
option of supplemental educational services from a state -approved provider. 


Corrective 

actions 


In Title I schools that miss AYP for a fourth year, districts also must implement at least one of the 
following corrective actions: replace school staff members who are relevant to the failure to make 
AYP; implement a new curriculum; decrease management authority at the school level; appoint an 
outside expert to advise the school; extend the school day or year; or restmcture the internal 
organization of the school. 


Restructuring 


In Title I schools that miss AYP for a fifth year, districts also must begin planning to implement at 
least one of the following restructuring interventions: reopen the school as a charter school; replace 
aU or most of the school staff; contract with a private entity to manage the school; turn over 
operation of the school to the state; or adopt some other major restructuring of the school’s 
governance. Districts must spend a year planning for restructuring and implement the school 
restructuring plan the following year (if the school misses AYP again for a sixth year). 


Highly 

qualified 

teachers 


All teachers of core academic subjects must be “highly qualified” as defined by NCLB and the state. 
To be highly qualified, teachers must have a bachelor’s degree, full state certification, and 
demonstrated competence in each core academic subject that they teach. Subject-matter competence 
may be demonstrated by passing a rigorous state test, completing a college major or coursework 
equivalent, or (for veteran teachers) meeting standards established by the state under a “high, 
objective uniform state standard of evaluation” (HOUSSE). 


Use of research 
based practices 


Schools must use effective methods and instructional strategies that are based on scienfificaUy-based 
research. 



























II. Title I Implementation Studies 



A. Key Evaluation Questions and Studies 

To answer questions of program implementation, the Department has relied on surveys of states, 
districts, schools, teachers, and parents as well as more in-depth case studies and analyses of state 
performance reports and other extant data sources. 

The key evaluation questions guiding the implementation studies are: 

• Whom does the Title I Part A program serve? How are the funds distributed? What does the 
money buy? 

• To what extent have states implemented and reported on the annual assessments in reading, 
mathematics, and science that are required under NCLB? Are students whom Title I is intended 
to benefit making progress toward meeting state academic achievement standards in reading and 
mathematics? 

• What are the reasons that schools do not make adequate yearly progress? How many and what 
types of schools are identified for improvement? What assistance is provided to schools and 
districts identified for improvement? 

• How many students are eligible to participate in school choice and supplemental education 
services, and how many actually do so? 

• How have states implemented the standards and procedures for teachers to demonstrate that 
they are “highly qualified teachers” and how many teachers meet these requirements? To what 
extent are teachers participating in professional development activities? 

The report draws on data from two Department evaluations of NCLB implementation, the National 
Longitudinal Study of NCLB, and the Study of State Implementation of Accountability and Teacher 
Quality Under NCLB, both of which collected data in the 2004-05 school year. The report also includes 
data from other evaluation studies, state performance reports, the National Assessment of Educational 
Progress, and other sources. 
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B. Key Findings 



These key findings are drawn from Volume II: Implementation (referenced at the bottom of this page*) 
which contains a more extensive discussion of the implementation study findings. Unless otherwise 
indicated, the key findings reported below describe the Tide I program nationally. AU findings reported 
were statistically significant at the 0.05 level. 

Title I Participants and Funding 

Title I funds go to 93 percent of the nation’s school districts and to 56 percent of all public schools. 

Most Title I funds (74 percent) go to elementary schools, and nearly three-fourths (72 percent) of Tide I 
pardcipants in 2004-05 were in pre-kindergarten though grade 6. Minority students accounted for two- 
thirds of Tide I participants. 

• Fueled by a growing use of Title I schoolwide programs (see Exhibit 2), the number of 
students counted as Title I participants has tripled over the past decade, rising from 
6.7 million in 1994-95 to 20.0 million in 2004-05. In 2004-05, 87 percent of Tide I 
pardcipants were in schoolwide programs. 

• The number of private school students participating in Title I has increased gradually 
over the past 20 years, to 188,000 in 2004-05, although it remains below the high of 213,500 
reached in 1980-81. Private school students typically received Tide I services from district 
teachers who traveled to the private school to serve students. Private school principals reported 
that districts usually consulted with private school representadves about Tide I services, although 
they indicated that professional development, parent involvement, and student assessment were 
not always covered in those consultadons. 

• Funding for Title I, Part A, has increased by 35 percent over the past seven years, after 
adjusdng for infiadon, from $9.5 billion in FY 2000 to $12.8 billion in FY 2007. 

• A majority of Title I funds were targeted to high-poverty districts and schools, but low- 
poverty districts and schools also received these funds. In 2004-05, about three-fourths 
(76 percent) of Tide I funds went to high— poverty schools (with 50 percent or more students 
eligible for free or reduced-price lunch). Low-poverty schools (with less than 35 percent of 
students eligible for free or reduced price lunch ) accounted for 14 percent of Tide I schools and 
received 6 percent of Tide I funds. 

• At the district level. Title I targeting has changed little since 1997-98, despite the intent 
of NCLB to target more funds to high-poverty school districts by allocating an 
increasing share of the funds through the Targeted Grants and Incentive Grants 
formulas. The share of funds appropriated through the Targeted and Incendve formulas rose 
from 1 8 percent of total Tide I funds in FY 2002 to 32 percent in FY 2004, while the less 
targeted Basic Grants formula declined from 85 percent to 57 percent of the funds. Despite 
these shifts, the share of funds actually received by the highest-poverty quardle of districts in 
2004-05 (52 percent) was similar to their share in 1997-98 (50 percent). 



Stullich, Stephanie, Elizabeth Eisner, Joseph McCrary. Report on the National Assessment of Title I, Volume I, 
Implementation of Title /, Washington, DC; U.S. Department of Education, Institute of Education Sciences, 2007. 
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Exhibit 2 

Number of Schoolwide Programs and Targeted 
Assistance Schools, 1 994-95 to 2004-05 



60,000 

50.000 

40.000 

30.000 

20.000 
10,000 

0 




1994-95 1996-97 1998-99 2000-01 2002-03 2004-05 



Exhibit reads: The number of schoolwide programs increased 
from 5,050 in 1994-95 (10 percent) to 31,782 in 2004-05 (58 
percent). 

Source: Consolidated State Performance Reports (for 50-52 states). 



• At the school level, the share of Title I funding for the highest-poverty schools also 
remained virtually unchanged since 1997-98, and those schools continued to receive 
smaller Title I allocations per low-income student than did low-poverty schools. The 

average Title I allocation in the highest-poverty Title I schools was $558 per low-income student 
in 2004-05, compared with $563 in 1997-98 (see Exhibit 3). The middle two poverty groups of 
schools, however saw statistically significant increases in their per-pupil funding. Low-poverty 
schools did not see a significant change in their share of funding, but they continued to receive 
larger Title I allocations per low-income student than did the highest-poverty schools ($763 vs. 
$558). 

• Most Title I funds were used for instruction, supporting salaries for teachers and 
instructional aides, providing instructional materials and computers, and supporting 
other instructional services and resources. In the 2004-05 school year, nearly three-fourths 
(73 percent) of district and school Title I funds were spent on instruction, 16 percent were used 
for instructional support, and another 1 1 percent were used for program administration and 
other support costs such as facilities and transportation. About half (49 percent) of local Title I 
funds were spent on teacher salaries and benefits, with an additional 1 1 percent going for teacher 
aides. 
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Exhibit 3 

Average Title I School Allocation Per Low-Income Student, 
by School Poverty Level, 1997-98 and 2004-05 
(in constant 2004-05 dollars) 



$ 1,000 -| 



11997-98 □2004-05 



$914 



$800 -| 

$600 -I $558 

$400 



l$763 



$611 



$671 




$200 H 



Highest Poverty Second Highest Second Lowest Lowest Poverty 
Schools Poverty Schools Poverty Schools Schools 

(75% to 100%) (50% to <75%) (35% to <50%) (0%to<35%) 



Exhibit reads: The average Title I allocation in the highest-poverty 
schools was $558 per low-income student in 2004-05, about the same 
as in 1997-98. 

* Indicates that the 2004-05 amount is significantly different from the 1997-98 amount] 
(p<.05). 

Sources: Study of Education Resources and Federal Funding (1997-98) (n=4,563 
schools); National Longitudinal Study of NCLB (2004-05) (n=8,566 schools). 



Student Achievement 

For both state assessment and NAEP results, recent achievement trends through 2004 or 2005 
are positive overall and for key subgroups, particularly in mathematics and at the elementary 
level. At this early stage of NCLB implementation — states, districts, and schools began to implement 
the NCLB provisions in 2002-03 — it is not possible to say whether the trends described below are 
attributable to NCLB, to other improvement initiatives that preceded it, or a combination of both. 

• In states that had three-year trend data available from 2002-03 to 2004-05, the percentage 
of students achieving at or above the state’s proficient level rose for most student 
subgroups in a majority of the states. For example, state reading assessments administered in 
the 4th grade or an adjacent elementary grade show achievement gains in elementary reading for 
low-income students in 28 out of 35 states (80 percent) that had trend data available for this 
subgroup (see Exhibit 4). Across all student subgroups examined, states showed achievement 
gains in 78 percent of the cases. Results for mathematics and for 8th grade show similar 
patterns. 
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Exhibit 4 






Student Achievement Trends on State Assessments in 4* -Grade Reading and Mathematics, 


1 from 2002-03 to 2004-05, by Student Subgroup 






Percentage cf States Shewing Increase in 


Predicted Percentage of States That 




Prepertien cf Students Performing At or 


Would Meet 100% Proficient Target, 




Above the State’s Proficient Level 


Based on Recent Rates of Change 




Reading 


Mathematics 


Reading 


Mathematics 


All students 


83% 


89% 


26% 


34% 


Low-income 


80% 


92% 


29% 


38% 


Black 


76% 


80% 


32% 


33% 


Hispanic 


80% 


89% 


33% 


37% 


White 


71% 


89% 


24% 


35% 


LEP 


77% 


86% 


38% 


35% 


Migrant 


76% 


81% 


39% 


42% 


Students with disabilities 


78% 


81% 


28% 


30% 


Average across all subgroups 


78% 


86% 


31% 


35% 


Exhibit reads: The proportion of all students performing at or above states’ “proficient” levels in 4th-grade 
reading (or another nearby elementary grade) increased between 2002-03 and 2004-05 in 83 percent of the 
states that had consistent trend data available; however, based on the rates of change that states achieved 
during that period, only 26 percent of the states would reach the goal of 100 percent proficient by 2013-14. 


Source: Consolidated State Performance Reports (n = 25 to 36 states). For states that did not consistendy assess students in 4th-grade 


reading and mathematics from 2002-03 to 2004-05, either 3'^‘^-grade 


or 5*-grade assessments were used. 





• Based on trend data for 36 states, most states would not meet the goal of 100 percent 
proficiency by 2013-14 unless the percentage of students achieving at the proficient level 
increased at a faster rate. For example, 29 percent of the states with consistent elementary 
reading assessment data for low-income students would meet the 100 percent goal by 2013-14 
for this subgroup if they sustained the same rate of growth that they achieved from 2002-03 to 
2004-05 (see Exhibit 4). Looking across eight different student subgroups (low-income, black, 
Hispanic, white, LEP, migrant, students with disabilities, and aU students), an average of 

31 percent of the subgroups within the 36 states would be predicted to reach 100 percent 
proficiency in 4th grade reading based on current growth rates. Only one state (Nebraska) 
would be predicted to reach 100 percent proficiency for aU subgroups and assessments that were 
included in this analysis. 

• Recent NAEP trends showed gains for 4th-grade students in reading, mathematics, and 
science, overall and for minority students and students in high-poverty schools, but 
trends for middle and high school students were mixed. For example, from 2000 to 2005, 
4th-grade black students gained 10 points in reading and Hispanic students gained 13 points, 
while in mathematics, black students gained 17 points and Hispanic students gained 18 points. 
Over the longer term, black and Hispanic students showed even larger gains in mathematics (33 
points and 26 points, respectively, from 1990 to 2005), but somewhat smaller gains in reading 
(eight points and seven points, respectively, from 1992 to 2005) (see Exhibit 5). 
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• Neither 8*- nor 12*-grade students made gains in reading or science achievement. 

Eighth-grade students made significant gains in mathematics, but not in reading or science. At 
the 12*-grade level, reading and science achievement in 2005 was unchanged from the preceding 
assessments (2002 for reading and 2000 for science) and showed significant declines from the 
first years those assessments were administered (1992 for reading and 1996 for science). Recent 
trend data for 12*-grade mathematics are not available. 

• State assessments and NAEP both provided some indications that achievement gaps 
between disadvantaged students and other students may be narrowing. For example, 
state assessments showed a reduction in the achievement gap between low-income students and 
all students in most states, typically a reduction of one to three percentage points. On the Trend 
NAEP, which has used a consistent set of assessment items since the 1970’s, achievement gains 
for black and Hispanic substantially outpaced gains made by white students, resulting in 
significant declines in black-white and Hispanic-white achievement gaps, but recent changes in 
achievement gaps often were not statistically significant. 

Under NCLB, high schools are held accountable for graduation rates, but methods for calculating 
graduation rates vary considerably across states. The averaged freshman graduation rate (calculated by 
the National Center for Education Statistics based on data from the Common Core of Data) is useful for 
providing a common standard against which state-reported graduation rates may be compared. The 
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median state graduation rate in 2004 was 84 percent based on state reports and 77 percent based on the 
averaged freshman graduation rate. 

• The recent trend in the averaged freshman graduation rate has been fairly steady, and 
the mean graduation rate in 2004 (75 percent) was slightly higher than in 1996 (73 
percent). However, these longitudinal data may not be stricdy comparable because of changes 
in reporting over time. 

Implementation of State Assessment Systems 

During the 2005-06 school year, all states administered assessments intended to meet NCLB 
requirements for reading and mathematics, and as of September 2007, 24 state assessment 
systems had been approved by the Department, through a peer review process, as meeting aU 
NCLB testing requirements. The remaining 28 states fell into one of two categories: approval 
expected (8), or approval pending (20). The eight states currentiy designated as “approval expected” 
submitted evidence indicating that their assessments were fully compliant with the statutory and 
regulatory requirements, but certain elements were not yet complete because of the nature of assessment 
development. For the 20 states designated as “approval pending,” the evidence submitted indicated that 
one or more fundamental components were missing or did not meet the statutory and regulatory 
requirements. Although science assessments are not required until 2007-08 under NCLB, three states 
had their general and alternate assessments in science approved ahead of schedule along with their 
reading and mathematics assessments. 

• Many state approaches to assessing English language proficiency (ELP) were stiU 
evolving as of 2004-05. Although aU states were implementing some kind of ELP assessment 
in 2004-05, 44 states indicated that they anticipated making revisions to their ELP assessments. 
Twenty states reported that their ELP assessment met NCLB requirements, 27 states planned to 
implement an ELP assessment that meets NCLB requirements by 2005-06, and five states had 
not decided which ELP assessment to use to meet NCLB requirements. 

• As of 2004-05, states varied in the extent to which they have met the requirement to 
annually assess 95 percent or more of their students across various subgroups. For the 

low-income student subgroup, 48 states assessed at least 95 percent of students in reading and 
49 states assessed at least 95 percent of students in math. For the students with disability 
subgroup, 45 states assessed at least 95 percent of students in reading and 46 states met the 
requirement in math. However, for the LEP student subgroup, 38 states met the requirement in 
reading and 45 states met the requirement in math. 

Accountability and Support for School Improvement 

States identified 12 percent (11,648) of aU schools for improvement for 2005-06 (based on test 
scores for 2004-05 and earlier). Title I schools accounted for 84 percent of all identified schools, 
and the 9,808 identified Title I schools represented 18 percent of aU Title I schools. About two- 
thirds (68 percent) of the identified Title I schools were in their first year or second year of 
improvement, with another 14 percent in corrective action and 19 percent in restructuring status. 

• Schools with high concentrations of poor and minority students were more likely to be 
identified than other schools, as were schools located in urban areas. Just over one-third 
of high-poverty schools (32 percent) and schools with high percentages of minority students 
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(31 percent) were identified schools in 2004-05, compared with 4 percent of schools with low 
concentrations of these students (see Exhibit 6). 



Exhibit 6 

Percentage of Schoois Identified for Improvement, 
By School Characteristics, 2005-06 




■ 75% or More ■ 75% or More ■ Urban 

□ 35% to <75% □ 25% to <75% □ Suburban 

■ Less than 35% ■ Less than 25% ■ Rural 



Exhibit reads: In 2005-06, 32 percent of high-poverty schools 
were identified for improvement, compared with 4 percent of 
low-poverty schools. 

Source: Study of State Implementation of Accountability and Teacher Quality 
Under NCLB (based on data reported by 51 states and between 80,812 and 
87,728 schools). 



Adequate Yearly Progress 

Three-fourths (75 percent) of all schools and districts met all applicable AYP targets in 2004-05 
testing. The number of all schools missing AYP (22,093) based on 2004-05 testing is nearly double the 
number of schools identified for improvement for 2005-06 (11,648). If many non-identified schools that 
did not make AYP in 2004-05 testing missed AYP again the following year, the number of identified 
schools could rise substantially in 2006-07. 

• Schools most commonly missed AYP for the achievement of all students and/ or multiple 
subgroups; only in a minority of cases did schools miss only one AYP target. Based on 
data from 39 states, among schools that missed AYP in 2004-05, 43 percent did not meet 
achievement targets for the “all students” group in reading, mathematics, or both and another 19 
percent missed AYP for the achievement of two or more subgroups (see Exhibit 7). Only 21 
percent missed AYP solely due to the achievement of a single subgroup. Four percent missed 
solely due to the “other academic indicator,” and 3 percent missed solely due to insufficient test 
participation rates. The remaining 10 percent of schools that missed AYP missed for other 
combinations of AYP targets. 
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Exhibit 7 

AYP Targets Missed by Schoois That 
Did Not Make Adequate Yearly Progress, 2004-05 
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Exhibit reads: In 2004-05 testing, 43 percent of schools missed AYP for 
the achievement of the “All Students” group in reading, mathematics, or 
both. 



Source: Study of State Implementation of Accountability and Teacher Quality Under 
NCLB (based on data from 39 states and 19,471 schools that missed AYP in these states). 



• Schools in states that had set more challenging proficiency standards than other states, 
as measured relative to NAEP, were less likely to make AYP and have further to go to 
reach the NCLB goal of 100 percent proficient. In states that had higher proficiency 
standards in 4th and 8th grade reading (based on using NAEP to benchmark the states against a 
common metric), 70 percent of schools made AYP in 2003-04, compared with 84 percent of 
schools in states that had lower proficiency standards. 

• NCLB required states to set starting points for the percentages of students achieving at 
the proficient level in order to measure progress towards the goal of 100 percent 
proficiency. States that had higher standards tended to have lower starting points and 
thus had further to go to reach 100 percent proficiency, compared with states that had set 
lower standards. For example, for 8th grade mathematics, states with higher proficiency 
standards had an average starting point of 1 6 percent, and therefore need to raise their 
percentage of students performing at the proficient level by 84 percentage points, while states 
with lower proficiency standards had an average starting point of 51 percent and need to raise 
their percent proficient by 49 percentage points (see Exhibit 8). 
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Exhibit 8 

improvement Needed to Reach 100 Percent Proficiency 
by 2013-14, by Level of Difficulty of State Academic 
Achievement Standards, for 8th Grade Mathematics 




Proficiency Proficiency Proficiency 

Standards Standards Standards 



Exhibit reads: States that had set higher proficiency standards 
than other states (measured relative to NAEP) had an average 
AYP starting point of 19 percent and needed to increase their 
percentage of students achieving at the proficient level by 81 
percentage points in order to reach NCLB’s goal of 100 percent 
proficiency by 2013-14. 

Note: States were required to set starting points for measuring progress towards 
the goal of 100 percent proficiency, based on the percentage of students achieving 
at the proficient level in 2001-02, either for the lowest-achieving student subgroup 
in the state or for the school at the 20th percentile of enrollment among schools 
ranked by their percent proficient (whichever is higher). 

Sources: Study of State Implementation of Accountability and Teacher Quality 
Under NCLB (n=34 states). Categorizations of states as having higher or 
lower proficiency standards are based on data from the National Center for 
Education Statistics (2007), Mapping 2005 State Proficiency Standards Onto 
the NAEP Scales (NCES 2007-482). Data on average starting points are from 
State Accountability Workbooks and State Educational Agency Websites. 



School Improvement Activities 

Fifteen states notified schools of the final determinations on whether or not they had been 
identified for improvement for 2004-05 (based on 2003-04 testing) before September 2004. Thirty 
states provided preliminary results by that time. NCLB regulations require states to notify schools and 
districts of their school improvement status prior to the beginning of the school year; this is important to 
enable districts with identified schools to notify parents of eligible students about their Title I choice 
options in a timely manner. 

• Identified schools were more likely to report needing assistance in a variety of specific 
areas than non-identified schools, and they also reported receiving more days of 
assistance than non-identified schools. For example, 80 percent of identified schools 
reported needing technical assistance to improve the quality of professional development, 
compared with 53 percent of non-identified schools. Similarly, 74 percent of identified schools 
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reported needing assistance to get parents more engaged in their child’s education, compared 
with 46 percent of non-identified schools. 

• Nearly one-third (30 percent) of elementary schools identified for improvement reported 
increasing the amount of instructional time in reading by more than 30 minutes per day 
in 2004-05, and 17 percent reported a similar increase in instructional time for 
mathematics. Non-identified schools less frequendy reported such increases. Fifty-five 
percent of identified secondary schools also more commonly reported increasing instructional 
time for low-achieving students in reading compared to 36 percent of non-identified schools. 

• Almost three-fourths of all schools offered extended-time instructional programs, and 
the percentage of students served through after-school programs doubled from 1997-98 
to 2004-05 (from 5 percent to 10 percent). In schools that implemented after-school programs, 
the programs provided an additional 134 hours of instruction annually, on average, or about a 12 
percent increase in instructional time for participating students. 

• Twenty-two percent of principals and 30 percent of elementary teachers in identified 
schools were not aware that their school had been identified as in need of improvement. 

Parents in a sample of eight urban school districts were less likely to know whether their child’s 
school had been identified as low-performing, compared with principals and teachers; however, 
parents in schools that had been identified for improvement were significantiy less likely than 
other parents to express satisfaction with their school. 

School Choice and Supplemental Educational Services 

Although more students were efigible to participate in the Title I school choice option than in 
supplemental educational services, a larger number of students actually participated in the 
supplemental services option. Based on district reports, more than twice as many students were 
eligible to transfer to another school under the Tide I school choice option in 2004-05 (5.2 million) as 
were eligible to receive supplemental services (2.4 million). However, nearly ten times as many students 
actually participated in the supplemental services option (446,000) as participated in the school choice 
option (48,000) in that year (see Exhibit 9). 

• Student participation in both Title I choice options has increased several fold since the 
first year of implementation of the NCLB choice provisions. Participation in the school 
choice option more than doubled over the three-year period from 2002-03 to 2004-05, while 
participation in supplemental services increased more than ten-fold. 
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Exhibit 9 

Number of Students Participating in Titie I Schooi Choice 
and Suppiementai Services, 2002-03 to 2004-05 




Exhibit reads: The number of students participating in Tide I 
school choice rose from 18,000 in 2002-03 to 48,000 in 2004-05. 

Source: Study of Title 1 Accountability Systems and Schooi Improvement 
Efforts (2002-03); National Longitudinal Study of NCLB and Study of State 
Implementation of Accountability and Teacher Quality Under NCLB (2003-04); 
Consolidated State Performance Reports (2004-05). 



• In a sample of nine urban districts*, African-American students had the highest 
participation rate of all racial and ethnic groups in Title I supplemental services and an 
above-average participation rate in Title I school choice (16.9 percent and 0.9 percent, 
respectively). Hispanic students, LEP students, and students with disabilities had relatively 
high participation rates in supplemental services and relatively low participations rates in school 
choice (see Exhibit 10). 

• Although nationally nearly all districts required to offer school choice and supplemental 
services reported that they notified parents about these options, a survey of eligible 
parents in eight urban school districts* found that many parents were unaware of these 
choice options. Among parents with a child eligible for the Title I school choice option, 27 
percent said they had received notification about this option from the school district, while 53 
percent of parents with a child eligible for supplemental services said they had been notified. 



An analysis of Title I choice options in nine large urban school districts provides more in-depth information about 
the characteristics of participating students in these districts; a survey of parents was also conducted in eight of the 
nine districts. Because this district sample was not nationally representative, findings cannot be generalized to the 
nation. 
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Exhibit 10 

Participation Rates for Title 1 School Choice and Supplemental Educational Services, 
By Student Subgroup, in Nine Large Urban School Districts, 2004-05* 




School Choice 


Supplemental Services 


White 


1.1% 


10.1% 


Black 


0.9% 


1 6.9% 


Hispanic 


0.4% 


1 1 .6% 


LEP 


0.3% 


13.1% 


Students with disabilities** 


0.4% 


14.6% 


Exhibit reads: In a sample of nine large urban school districts, 1.1 percent of eligible white students 
participated in the Title I school choice option. 


* Data for one district are for 2003-04. 

** Data for students with disabilities are based on seven districts. 








Source: National Longitudinal Study of NCLB, Analysis of Title I Choice Options in Nine Urban Districts. 





• Most participating students received supplemental services from a private provider, but 
school districts and public schools also served a substantial share of participants. Private 
firms accounted for 86 percent of approved providers in May 2007, while school districts and 
public schools accounted for only 11 percent of providers. However, earlier data, from 2003-04, 
indicate that school districts and public schools serve a relatively high proportion of participating 
students (40 percent). 

• Based on a survey of supplemental service providers in 16 school districts*, services were 
most often provided at the student’s school. Sixty-one percent of providers in the 16 
districts reported that services were always or often provided at the student’s school; other 
locations were the local office of the provider (26 percent), libraries or community centers (19 
percent), and over the internet (11 percent). 

• Services are provided both through one-on-one tutoring and through group instruction. 

In the 1 6 districts, over half of the providers said that they often or always served students one- 
on-one (52 percent) or in small groups (52 percent), while 34 percent said services were often or 
always provided in large groups. Services were provided for an average of 57 hours per student 
per year in those districts, and students attended an average of 78 percent of the sessions. 
Maximum funding per student, as reported by districts, was $1,434 in 2004-05. 

• States reported that they were working to develop and implement systems for monitoring 
and evaluating the performance of supplemental service providers, but, as of early 2005, 
15 states had not established any monitoring process, 25 states had not yet established 
any standards for evaluating provider effectiveness, and none had finalized their 
evaluation standards. Seventeen states said they wiU evaluate the provider services against 
student achievement on state assessments. One of these states planned to use a matched control 



A survey of 125 supplemental services providers in 16 school districts provides additional information about the 
services provided in these districts. Because the district sample was not nationally representative, findings cannot be 
generalized to the nation. 
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group. The most common approaches that states have implemented to monitor providers are 
surveying the districts about provider effectiveness (25 states) and using providers’ reports on 
student-level progress (18 states). 

• Although NCLB assigns states the responsibility for monitoring and evaluating 
providers, a survey of providers in 16 districts found that the providers reported more 
frequent monitoring by districts than by states. For example, over half (51 percent) of the 
providers said that district staff observed supplemental service sessions at least a few times a 
year, compared with only 22 percent that experienced this frequency of observations by state 
staff. 

Teacher Quality and Professional Development 

The large majority of teachers across the country have been designated as “highly qualified” 
under NCLB. According to state-reported data for 50 states, 91 percent of classes were taught by 
highly qualified teachers in 2004-05. Principal and teacher reports provide somewhat lower estimates of 
the percentage of classes taught by highly qualified teachers. This is partiy due to a si 2 eable percentage 
of teachers not knowing their “highly qualified” status. For example, 74 percent of teachers reported 
that they were considered highly qualified under NCLB, 23 percent said they did not know their status, 
and 2 percent said they were not highly qualified. Special education teachers and secondary mathematics 
teachers were more likely to report that they were considered not highly qualified under NCLB than 
were general elementary teachers and secondary English teachers. 

• Students in schools that have been identified for improvement were more likely to be 
taught by teachers who said they were not highly qualified than were students in non- 
identified schools. For example, only one percent of elementary teachers in non-identified 
schools said they were considered not highly qualified, compared with 5 percent in schools that 
were in the first or second year of being identified for improvement, 8 percent in schools in 
corrective action, and 6 percent of schools in restructuring. 

• Among teachers who said they were highly qualified under NCLB, those in high-poverty 
schools had less experience and were more likely to be teaching out-of-field, compared 
with their peers in low-poverty schools. For example, 12 percent of highly qualified teachers 
in high-poverty schools had fewer than three years of teaching experience, compared with 5 
percent of highly qualified teachers in low-poverty schools. Similarly highly qualified secondary 
English and mathematics teachers in high-poverty schools were less likely to have a degree in the 
field that they teach (41 percent compared with 52 percent in low-poverty schools). 

• High-poverty and high minority districts were more likely than other districts to say that 
competition with other districts was a barrier to attracting highly qualified teachers, and 
were also more likely to report using financial incentives and alternative certification 
routes in an effort to overcome these barriers. For example, 29 percent of high-poverty 
districts and 7 5 percent of high-minority districts reported using financial incentives, compared 
with 18 percent of low-poverty districts and 12 percent of low-minority districts. 

• Most teachers reported receiving some professional development in reading and math 
content and instructional strategies, but fewer than one-quarter of the teachers 
participated in such training for more than 24 hours over the 2003-04 school year and 
summer. For example, 90 percent of elementary teachers participated in at least one hour of 
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professional development focused on instructional strategies for teaching reading, but only 20 
percent participated for more than 24 hours over the 2003-04 school year and summer (see 
Exhibit 11). 

• Teachers in high -poverty schools were more likely to participate in professional 

development focused on reading and mathematics than were teachers in low-poverty 
schools. For example, 53 percent of secondary English teachers in high-poverty schools 
reported participating in professional development focused on in-depth study of topics in 
reading or English compared with 36 percent of their colleagues in low-poverty schools. 



Exhibit 11 

Percentage of Teachers Participating in Professionai Deveiopment 
(PD) Focused on Instructionai Strategies for Reading and 
Mathematics, 2003-04 



100 % -| 
80% - 
60% 
40% H 
20% 

0% -h 




Elementary Secondary 
Teachers English 

(n=4,007) Teachers 

(n=1,740) 



Elementary Secondary 
Teachers Math 
(n=3,994) Teachers 

(n=1 ,580) 



PD in Reading 



PD in Mathematics 



Exhibit reads: Twenty percent of elementary teachers reported that they received 
more than 24 hours of professional development focused on instructional strategies 
for teaching reading during the 2003-04 school year. 



Source: National Longitudinal Study of NCLB, Teacher Survey. 
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III. Title I Impact Studies 



A central principle of NCLB is that states, districts, schools, and teachers adopt instructional practices 
backed by evidence of effectiveness from scientifically based research. This principle has created a 
demand for rigorous evaluation evidence currentiy unavailable for many education programs and subject 
areas. For this reason, the Department’s evaluation strategy for Title I features a strong emphasis on 
evaluation studies that are designed to produce rigorous scientific evidence on the effectiveness of 
specific education programs and practices that are critical to the effective use of Title I funds. 

At the second meeting of the Independent Review Panel on March 17-18, 2003, presentations were 
made by reading and mathematics experts on what we know and need to know in these areas to raise 
student achievement. Ultimately, three large-scale evaluations have been undertaken. Findings from the 
first study, examining the effects of remedial reading programs for S''* and 5* graders, are included in 
Volume II, Closing the Heading Gap, Findings from a Randomised Trial of Four Reading Interventions for Striving 
Readers. Impacts on student achievement at the end of the intervention year, as well as impacts one year 
after the interventions ended, can be found below in Section A. Closing the Reading Gap. Evaluations 
of the effectiveness of reading comprehension interventions for 5* graders and of early elementary math 
curricula were started during the 2006-07 school year. A full description of the two latter evaluations are 
described below in Section B. Future Reports. The rationales for these three large-scale evaluations of 
specific interventions are described briefly below in Sections A and B. 

A. Closing the Reading Gap 

This evaluation examines four widely used programs for elementary school students with reading 
problems. The programs are Corrective Reading, Failure Free Reading, Spell Read P.A.T., and Wilson 
Reading, all of which were hypothesried to be more intensive and delivered by better-trained teachers 
than the reading instmction typically provided in public schools. The programs incorporate explicit and 
systematic instruction in the basic reading skills in which struggling readers are frequently deficient. 
Corrective Reading, Spell Read P.A.T., and Wilson Reading were implemented to provide word-level 
instruction, whereas Failure Free Reading focused on building reading comprehension and vocabulary in 
addition to word-level skills. Recent reports from small-scale research and clinical studies provide some 
evidence that the reading skills of students with severe reading difficulties in late elementary school can 
be substantially improved by providing, for a sustained period of time, the kinds of skillful, systematic, 
and explicit instruction that these programs offer * . 

Conducted just outside Pittsburgh, Pennsylvania, in the Allegheny Intermediate Unit (AIU), the 
evaluation has explored the extent to which the four reading programs can affect both the word-level 
reading skills (phonemic decoding, fluency, accuracy) and reading comprehension of students in grades 
three and five who were identified as struggling readers by their teachers and by low test scores. 



* Torgesen, J.K. “Recent discoveries from Research on Remedial Interventions for Children with Dyslexia.” In M. 
Snowling and C. Hulme, eds., The Science of Reading. Oxford: Blackwell Publishers, 2005. 
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Ultimately, it provides educators with rigorous evidence of what could happen in terms of reading 
improvement if intensive, smaU-group reading programs Uke the ones in this study were introduced in 
many schools. 

Key Evaluation Questions and Study Design 

This study is a large-scale, longitudinal evaluation comprising two main elements. The first element of 
the evaluation is an impact study designed to address the following questions: 

• What is the impact of being in any of the four remedial reading interventions, considered as 
a group, relative to the instruction provided by the schools? What is the impact of being in 
one of the remedial reading programs that focuses primarily on developing word-level skills, 
considered as a group, relative to the instruction provided by the schools? What is the 
impact of being in each of the four particular remedial reading interventions, considered 
individually, relative to the instruction provided by the schools? 

• Do the impacts of the interventions vary across students with different baseline 
characteristics? 

• To what extent can the instruction provided in this study close the reading gap and bring 
struggling readers within the normal range, relative to the instruction provided by their 
schools? 

To answer these questions, we based the impact study on a scientifically rigorous design — an 
experimental design that uses random assignment at two levels: (1) 50 schools from 27 school districts 
were randomly assigned to one of the four interventions; and (2) within each school, eligible children in 
grades three and five were randomly assigned to a treatment group or to a control group. Students 
assigned to the intervention group (treatment group) were placed by the program providers and local 
coordinators into instructional groups of three students. They received supplemental reading instruction 
in these small groups in addition to regular reading instruction they would have usually received — unless 
they were puUed out of class during their regular reading instruction time. Students in the control groups 
received the same instruction in reading that they would have ordinarily received. Children were defined 
as eligible if they were identified by their teachers as struggling readers and if they scored at or below the 
30th percentile on a word-level reading test and at or above the 5th percentile on a vocabulary test. 

From an original pool of 1,576 third and fifth grade students identified as struggling readers, 1,502 were 
screened, and 1,042 met the test-score criteria. Of these eligible students, 779 were given permission by 
their parents to participate in the evaluation, and 772 were randomly assigned — 558 to the treatment 
group and 214 to the control group. 

The second element of the evaluation is an implementation study that has two components: (1) an 
exploration of the similarities and differences in reading instruction offered in the four interventions; and 
(2) a description of the regular instruction that students in the control group received in the absence of 
the interventions, and of the regular instruction received by the treatment group beyond the 
interventions. 

Test data and other information on students, parents, teachers, classrooms, and schools were collected 
several times over a two-year period. Key data collection points include the period just before the 
interventions began, when baseline information was collected, and the periods immediately after and one 
year after the interventions ended, when follow-up data were collected. 
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The Interventions 



New instructional programs were not designed for this evaluation. Either parts or all of four existing 
and widely used remedial reading instructional programs were employed. The programs are classified as 
either mrd level or word level plus comprehension, '^ord level interventions include methods that focus on 
improving word-level reading skills so that they no longer limit children’s ability to comprehend text. 
Such methods devote the majority of their instructional time to establishing phonemic awareness, 
phonemic decoding skills, and word and passage reading fluency. Methods in this classification 
sometimes include activities to check comprehension (such as asking questions and discussing the 
meaning of what is read), but this instruction is incidental to the primary focus on improving word-level 
reading skills. The bulk of instructional and practice time in methods included within this classification 
is focused on building children’s ability to read text accurately and fluentiy. The second intervention 
classification — referred to as word level plus comprehension — includes methods that more evenly balance 
instructional time between activities to build word-level skills and activities devoted to building 
vocabulary and reading comprehension strategies. These interventions include extended activities that 
are designed to increase comprehension and word knowledge (vocabulary), and these activities would 
take roughly the same amount of instructional time as the activities designed to increase word reading 
accuracy and fluency. 

• Spell Read Phonological Auditory Training (P.A. T.) provides systematic and explicit 
fluency-oriented instruction in phonemic awareness and phonics, along with every-day 
experiences in reading and writing for meaning. The phonemic activities include a wide variety of 
specific tasks focused on specific skill mastery and include, for example, building syllables from 
single sounds, blending consonant and vowel sounds, and analy 2 ing or breaking syllables into 
their individual sounds. Each lesson also includes reading and writing activities intended to help 
students apply their phonically based reading skills to authentic reading and writing tasks. The 
Spell Read intervention had originally been one of the two “word-level plus comprehension” 
interventions, but after the time-by-activity analysis, we determined that it was more 
appropriately classified as a “word-level” intervention. Because the word-level instructional 
content in Spell Read is more structured than the instruction designed to building reading 
comprehension, the relatively short instructional sessions in this study led to a different balance 
of word-level and comprehension instruction than was anticipated. 

• Corrective Reading xxses scripted lessons that are designed to improve the efficiency of 
instruction and to maximize opportunities for students to respond and receive feedback. The 
lessons involve explicit and systematic instructional sequences, including a series of quick 
tasks that are intended to focus students’ attention on critical elements for successful word 
identification (phonics and phonemic analysis), as well as exercises intended to build rate and 
fluency through oral reading of stories that have been constructed to counter word-guessing 
habits. Although the Corrective Reading program does have instructional procedures that 
focus on comprehension, they were originally designated as a “word-level intervention,” and 
the developer was asked not to include these elements in this study. 

• Wilson Reading uses direct, multi-sensory, structured teaching based on the Orton- 
GilUngham methodology. The program is based on 10 principles of instruction, some of 
which involve teaching fluent identification of letter sounds; presenting the structure of 
language in a systematic, cumulative manner; presenting concepts in the context of 
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controlled as well as noncontrolled text; and teaching and reinforcing concepts with visual- 
auditory-kinesthetic-tactile methods. Similar to Corrective Reading, the Wilson Program has 
instructional procedures that focus on comprehension and vocabulary, but since Wilson 
Reading was originally designated as a “word-level” intervention, the developer was asked 
not to include these in this study. 

• Failure Free Reading uses a combination of computer-based lessons, workbook exercises, 
and teacher-led instruction to teach sight vocabulary, fluency, and comprehension. The 
program is designed to have students spend approximately one-third of each instructional 
session working within each of these formats, so that they are not taught simultaneously as a 
group. UnUke the other three interventions in this study. Failure Free does not emphasi 2 e 
phonemic decoding strategies. Rather, the intervention depends upon building the student’s 
vocabulary of “sight words” through a program involving multiple exposures and text that is 
engineered to support learning of new words. Students read material that is designed to be of 
interest to their age level while also challenging their current independent and instructional 
reading level. Lessons are based on story text that is controlled for syntax and semantic 
content. 

The interventions provided instruction to students in the treatment group from November 2003 through 
May 2004. During this time students received, on average, about 90 hours of instruction, which was 
delivered five days a week to groups of three students in sessions that were approximately 55 minutes 
long. Instruction was provided by teachers who were recruited from participating schools on the basis 
of experience and characteristics and skiUs relevant to teaching struggling readers. They received, on 
average, nearly 70 hours of training and professional development support during the intervention year. 
According to an examination of videotaped teaching sessions, instruction was judged to be faithful to 
each intervention model. The program providers themselves also rated the teachers as generally above 
average in both their teaching skiU and fidelity to program requirements. 

Measures of Reading Ability 

Seven measures of reading skill were administered several times for the evaluation to assess student 
progress in learning to read. These measures assessed phonemic decoding, word reading accuracy, text 
reading fluency, and reading comprehension. 

Phonemic Decoding 

• Word Attack (WA) subtest from the Woodcock Reading Mastery Test-Revised (WRMT-R) 

• Phonemic Decoding Efficiency (PDE) subtest from the Test of Word Reading Efficiency 
(TOWRE) 

Word Reading Accuracy and Fluency 

• Word Identification (WI) subtest from the WRMT-R 

• Sight Word Efficiency (SWE) subtest from the TOWRE 

• Oral Reading Fluency subtest from Edformation, Inc. This report refers to the reading 
passages as “AIMSweb” passages, which is the term used broadly in the reading practice 
community. 

Reading Comprehension 

• Passage Comprehension (PC) subtest from the WRMT-R 
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• Passage Comprehension from the Group Reading Assessment and Diagnostic Evaluation 
(GRADE) 

The scores analyzed in this report are from the tests near the end of the 2004-2005 school year, one year 
after the interventions ended. Results from analyzing the scores on these tests from one year earlier — 
the end of the intervention year are presented in the National Assessment of Tide I Interim Report.^ 

In addition to analyzing data from seven reading tests administered by the study, Pennsylvania System of 
School Assessment (PSSA) reading and mathematics scores for each student were analyzed and included 
in the results. Students in the evaluation sample took these standards-based PSSA tests from late March 
to early April of the 2003-2004 school year, the year during which the interventions took place. 

Key Findings 

These key findings are drawn from Volume II: Closing the Heading Gap, bindings from a Kandomi^ied Trial of 
Four Heading Interventions for Striving Headers^ AH findings reported were statistically significant at the 0.05 
level. 

Characteristics of Students in the Evaluation 

The characteristics of the students in the evaluation sample are shown in Exhibit 12. 

• About 45 percent of the students qualified for free or reduced-price lunches. In addition, about 
28 percent were African American, and 72 percent were white. Fewer than two percent were 
Hispanic. Roughly 32 percent of the students had a learning disability or other disability. 

• On average, students in the evaluation sample scored about one-half to one standard deviation 
below national norms (mean 100 and standard deviation 15) on measures used to assess their 
ability to decode words. 

• This sample, as a whole, was substantially less impaired in basic reading skills than most other 
samples assessed in previous research with older reading disabled students (Torgesen 2005). 



^ Torgesen, Joseph, David Myers, Allen Schirm, Elizabeth Stuart, Sonya Vartivarian, Wendy Mansfield, Fran 
Stancavage, Donna Durno, Roseanne Javorsky, and Cinthia Haan. National Assessment of Title I Interim Report to 
Congress: Volume II: Closing the Reading Gap, First Year Findings from a Randomized Trial of Four Reading 
Interventions for Striving Readers, 2006. 

^ Torgesen, Joseph, Florida Center for Reading Research; Allen Schirm, Laura Castner, Sonya Vartivarian, Wendy 
Mansfield, Mathematic Policy Research; David Myers and Fran Stancavage, American Institutes for Research; 
Donna Durno and Rosanne Javorsky, Allegheny Intermediate Unit; and Cinthia Haan, Haan Foundation. Report on 
the National Assessment of Title I, Volume II, Closing the Reading Gap: Findings from a Randomized Trial of Four 
Reading Interventions for Striving Readers, Washington, DC; U.S. Department of Education, Institute of Education 
Sciences, 2007. 
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Exhibit 12 



Baseline Characteristics of the Analysis Sample 
3rd Grade and 5th Grade 









Grade Level 








Baseline Means 


Combined 


3rd 




5th 




student Characteristics 
















Age 


9.7 


8.7 




10.7 




Male (%) 


54 


52 




56 




Hispanic (%) 


2 




2 






1 




Race— White (%) 


72 


70 




74 




Race--African American (%) 


28 


30 
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Race— Other (%) 


a 


a 






a 




Family income less than $30,000 (%) 


49 


48 




49 




Family income between $30,000 and $60,000 (%) 


35 


34 




36 




Family income over $60,000 (%) 


16 


18 




15 




Eligible for Free or Reduced Price Lunch (%) 


45 


45 




46 




Has any learning or other disability (%) 


32 


34 




30 




Mother has bachelor's degree or higher (%) 


13 


13 




13 






Standard 




Standard 




Standard 




Reading Tests 


Score 


Percentile 


Score 


Percentile 


Score 


Percentile 


Screening Tests 

TOWRE Sight Word Efficiency 


84.3 


15 


84.4 


15 


84.1 




15 


TOWRE Phonemic Decoding Efficiency 


83.0 


13 


85.6 


17 


80.6 




10 


Peabody Picture Vocabulary Test-Revised 


94.7 


36 


94.4 


35 


95.0 




37 


Baseline Tests 

WRM Word Identification 


88.7 


23 


88.7 


22 


88.7 




23 


TOWRE Phonemic Decoding Efficiency 


83.2 


13 


85.5 


17 


81.0 




10 


WRM Word Attack 


92.8 


32 


92.4 


31 


93.2 




33 


TOWRE Sight Word Efficiency 


85.3 


16 


86.6 


19 


84.2 




15 


AIMSWeb (Raw score) 


NA 


NA 


40.9 


NA 


77.0 




NA 


WRM Passage Comprehension 


92.1 


30 


91.6 


29 


92.6 




31 


GRADE 


88.8 


23 


86.1 


18 


91.2 




28 


Woodcock Johnson Spelling 


89.8 


25 


88.5 


22 


90.9 




27 


Woodcock Johnson Calculation 


94.8 


36 


95.4 


38 


94.2 




35 


Sample Size 


729 


329 




400 





Note: Weights used to account for differential randomization probabilities and nonresponse. 

Note: All standard scores have mean 100 and standard deviation 15, except for CTOPP and Clinical Evaluation of Language 
Fundamentals-IV, which have mean 10 and standard deviation 3. Standard scores unavailable for the Aimsweb test. 

Note: The percentile score shown for each test is the percentile corresponding with the mean standard score. 



a Values suppressed to protect student confidentiality. 
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Impact Findings 

In the first report from the evaluation (Torgesen et al. 2006), impacts on reading test scores were 
presented for the end of the intervention year, when the students in the evaluation were third and fifth 
graders. That report presented findings on student outcomes at the end of the intervention year. The 
second (and last) report (Torgesen et al. 2007) from the evaluation, presents estimates of impacts on 
reading test scores as of the end of the following year, when most of the students were fourth- and sixth- 
graders. Student outcomes are referred to as their “grade cohorts,” that is the students’ grade level when 
they entered the evaluation. 

For purposes of this summary, the focus is on the impact of being randomly assigned to receive any 
intervention compared to receiving the instruction that would normally be provided. These findings are 
the most robust because of the larger sample skes. The full report also estimates impacts for each of the 
interventions and various student subgroups. Key findings are as follows: 

• The interventions improved some reading skills. For the third-grade cohort, the four 
interventions combined had positive impacts on phonemic decoding, word reading accuracy 
and fluency, and reading comprehension, although impacts were not detected for all 
measures of accuracy and fluency or comprehension (see Exhibit 13). For the fifth-grade 
cohort, the four interventions combined improved phonemic decoding on one measure, but 
led to a small reduction in oral reading fluency. The three word-level interventions 
combined had similar impacts to those for all four interventions combined, although they 
did not have an impact on either measure of comprehension for students in the third grade 
cohort. There were impacts on both measures of phonemic decoding for students in the 
fifth-grade cohort. For students in the third-grade cohort. Failure Free Reading (the only 
word level plus comprehension program) had an impact on one measure of phonemic 
decoding, two of the three measures of word reading accuracy and fluency, and one measure 
of comprehension. However, this intervention did not have any impacts for students in the 
fifth-grade cohort. 

• The interventions did not improve PSSA scores. For the third-grade cohort, we did not 
detect significant impacts of the four interventions combined on reading and mathematics 
test scores from the Pennsylvania System of School Assessment (see Exhibit 14). For the 
fifth-grade cohort, the four interventions combined lowered the reading and mathematics 
scores. 

• Younger students benefited more. The interventions generally helped students in the 
third-grade cohort more than students in the fifth-grade cohort (see Exhibits 13 and 14). 
However, the interventions did not consistendy benefit any one subgroup within each grade 
level more than another subgroup. 
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• The interventions narrowed some reading gaps. The “reading gap” describes the extent 
to which the average 3*^'^ or 5* grade student in the intervention group or in the control 
group is lagging behind the average 3 ’^'^ or 5* grade student in the population. The four 
interventions combined generally narrowed the reading gap for students in the intervention 
groups compared with students in the control group for the third-grade cohort. Being in 
one of the interventions reduced the reading gap in Word Attack skills by about two-thirds 
for students in the third-grade cohort. On other word-level tests and a measure of reading 
comprehension, the interventions reduced the gap for students in the third-grade cohort by 
about one-sixth to one-third. For students in the fifth grade cohort, the interventions 
reduced the gap in Word Attack skills by one-half (see Exhibit 15). 

The key findings presented in this report for the seven tests administered for this study one year 
after the interventions ended are similar to the findings from the end of the intervention year. In 
the earlier report (Torgesen et al. 2006) the four interventions combined and the three word- 
level interventions had impacts for student in the third-grade cohort on phonemic decoding, 
word reading accuracy and fluency, and reading comprehension. There were fewer significant 
impacts for students in the fifth-grade cohort than for students in the third-grade cohort. 
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Exhibit 13 



Impacts on Reading Test Scores for 3rd and 5th Grade Cohorts 
One Year After the Intervention Year 



Grade 3 Cohort 


Baseline 


All Interventions 




Word-level Interventions 


Failure Free Reading 


Spell Read 


Wilson Reading 


Corrective Reading 


Control 

Gain 


ABCD 

Impact 




Control 

Gain 


BCD 

Impact 




Control 

Gain 


A 

Impact 


Control 

Gain 


B 

Impact 


Control 

Gain 


c 

Impact 


Control 

Gain 


D 

Impact 


Word Attack 


92.4 


-0.3 


5.3 * 




0.3 


5.5 * 




-2.2 


4.9 * 


-0.3 


5.4 * 


0.5 


5.8 * 


0.7 


5.2 * 


TOWRE PDE 


85.5 


1.1 


4.0 * 




1.0 


4.9 * 




1.3 


1.3 


3.4 


4.9 * 


1.8 


4.1 * 


-2.3 


5.5 * 


Word Identification 


88.7 


-0.1 


2.3 * 




0.0 


2.5 * 




-0.4 


1.8 


1.8 


0.7 


-2.2 


4.1 * 


0.6 


2.6 * 


TOWRE SWE 


86.6 


3.5 


1.7 * 




3.7 


1.6 * 




2.7 


2.0 * 


4.0 


0.9 


2.8 


2.6 * 


4.3 


1.4 


AIMSweb 


40.9 


33.7 


5.3 




33.4 


4.5 




34.4 


7.9 * 


30.1 


6.0 * 


31.9 


3.6 


38.3 


3.9 


Passage Comprehension 


91.6 


-0.3 


2.1 * 




0.3 


1.3 




-2.1 


4.4 * 


1.2 


0.1 


-2.5 


3.5 


2.3 


0.3 


GRADE 


86.1 


-7.5 


1.0 




-6.9 


0.4 




-9.3 


2.8 


-10.0 


2.1 


-10.4 


0.1 


-0.1 


-1.1 


Sample Size 


329 




329 






240 






89 




91 




70 




79 






All Interventions 




Word-level Interventions 


Failure Free Reading 


Spell Read 


Wilson Reading 


Corrective Reading 






Control 


ABCD 




Control 


BCD 




Control 


A 


Control 


B 


Control 


C 


Control 


D 


Grade 5 Cohort 


Baseline 


Gain 


Impact 




Gain 


Impact 




Gain 


Impact 


Gain 


Impact 


Gain 


Impact 


Gain 


Impact 


Word Attack 


93.2 


1.5 


2.7 * 


# 


1.8 


3.8 * 




0.4 


-0.8 # 


0.0 


3.5 


2.3 


7.8 * 


3.1 


0.2 # 


TOWRE PDE 


81.0 


5.3 


1.7 




5.2 


2.4 * 




5.4 


-0.3 


4.9 


3.2 


4.2 


2.6 


6.6 


1.4 


Word Identification 


88.7 


3.0 


-0.6 


# 


3.4 


-0.6 


# 


1.7 


-0.6 # 


1.5 


0.1 


4.3 


0.0 # 


4.3 


-1.9 # 


TOWRE SWE 


84.2 


3.0 


1.4 




3.1 


1.4 




3.0 


1.5 


1.5 


3.4 * 


2.7 


1.1 


5.0 


-0.4 


AIMSweb 


77.0 


30.9 


-3.9 * 


# 


30.7 


-3.9 * 


# 


31.6 


-4.1 # 


26.5 


-3.3 # 


29.7 


-3.0 


35.9 


-5.3 


Passage Comprehension 


92.6 


-0.4 


-1.1 


# 


-0.8 


-0.7 




0.8 


-2.5 # 


-2.8 


-0.9 


-1.6 


0.9 


1.9 


-2.1 


GRADE 


91.2 


-3.5 


0.7 




-4.7 


1.2 




0.2 


-0.9 


-4.8 


-1.1 


-8.9 


4.7 


-0.4 


0.0 


Sample Size 


400 




400 






272 






128 




100 




88 




84 



Note: The Failure Free, Spell Read, Wilson Reading, and Corrective Reading interventions are labeled A, B, C, and D, respectively. These labels are arbitrary and not related to performance. 
ABCD is the label for the four interventions combined and BCD is the label for the three word-level intervention combined. 

Note: Raw scores were analyzed for the AIMSweb, and standard scores were analyzed for all other tests. 

Note: According to the first row of estimates, students in the third-grade cohort achieved an average standardized Word Attack score of 92.4 at “baseline,” that is, shortly after the beginning of 
third grade — the intervention year. For the Failure Free Reading intervention, the average standardized Word Attack score one year after the intervention year fell by 2.2 points from the baseline 
score for the students in the control group (the “control gain”). Also one year after the intervention year, the average score for the students in the treatment group for the Failure Free Reading 
intervention was 4.9 points higher than for the students in the control group (the “impact”), a difference that is statistically significant, as indicated by the asterisk. According to the columns for 
“All Interventions,” the average score for the control group was 0.3 points lower than the baseline score, and the average score for the treatment group was 5.3 points higher than the average for 
the control group, a statistically significant difference. 

* Impact is statistically significant at the 0.05 level. 

# Impact is statistically different from the 3rd grade cohort impact at the 0.05 level. 
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Exhibit 14 



Impacts on PSSA Reading and Math Scores for 3rd and 5th Grade Cohorts 
Late March/Early April of the Intervention Year 







Word-level 


Failure Free 






Corrective 




All Interventions 


Interventions 


Reading 


Spell Read 


Wilson Reading 


Reading 




ABCD 


BCD 


A 


B 


C 


D 


Grade 3 Cohort 


Impact 


Impact 


Impact 


Impact 


Impact 


Impact 


PSSA Reading 


-15.6 


-3.8 


-51.1 * 


-39.9 


52.5 


-23.8 


PSSA Math 


20.2 


14.2 


38.4 


-15.5 


56.6 * 


1.4 


Sample Size 


329 


240 


89 


92 


71 


77 






Word-level 


Failure Free 






Corrective 




All Interventions 


Interventions 


Reading 


Spell Read 


Wilson Reading 


Reading 




ABCD 


BCD 


A 


B 


C 


D 


Grade 5 Cohort 


Impact 


Impact 


Impact 


Impact 


Impact 


Impact 


PSSA Reading 


-27.3 * 


-25.3 


-33.4 * 


-30.0 


-23.8 


-22.1 


PSSA Math 


-28.8 * # 


-34.0 * # 


-13.4 


-20.1 


-56.4 * # 


-25.4 * 


Sample Size 


408 


280 


128 


102 


92 


86 



Note: The Failure Free, Spell Read, Wilson Reading, and Corrective Reading interventions are labeled A, B, C, and D, respectively. These 
labels are arbitrary and not related to performance. ABCD is the label for the four interventions combined and BCD is the label for the three 
word-level intervention combined. 



Note: According to the first row of estimates, students in the third-grade cohort assigned to the Failure Free Reading intervention achieved a 
standardized score on the PSSA Reading test that was 51.1 points lower than the average score achieved by the students in the control group, a 
statistically significant difference, as indicated by the asterisk. The average standardized score for students participating in any intervention was 
15.6 points lower than the average score for students assigned to a control group, a difference that is not statistically significant. 

* Impact is statistically significant at the 0.05 level. 

# Impact is statistically different form the 3rd grade impact at the 0.05 level. 
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Exhibit 15 



Relative Gap Reduction (RGR): All Interventions Combined 
One Year After the Intervention Year 



Grade 3 Cohort 


Average at 
Baseline 


Gap at 

baseline (Std. 
Units) 


Average at follow-up 

Intervention Control 

Group Group 


Gap at follow-up (Std. 
Units) 

Intervention Control 

Group Group 


Impact 


RGR 


Word Attack 


92.4 


0.50 


97.4 


92.1 


0.17 


0.53 


5.3 * 


0.68 


TOWRE PDE 


85.5 


0.97 


90.5 


86.6 


0.63 


0.90 


4.0 * 


0.29 


Word Identification 


88.7 


0.76 


90.9 


88.6 


0.61 


0.76 


2.3 * 


0.20 


TOWRE SWE 


86.6 


0.90 


91.7 


90.0 


0.55 


0.67 


1.7 * 


0.17 


AIMSweb 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


Passage Comprehension 


91.6 


0.56 


93.4 


91.3 


0.44 


0.58 


2.1 * 


0.24 


GRADE 


86.1 


0.93 


79.6 


78.6 


1.36 


1.42 


I.O 


0.05 



Grade 5 Cohort 


Average at 
Baseline 


Gap at 

baseline (Std. 
Units) 


Average at follow-up 

Intervention Control 

Group Group 


Gap at follow-up (Std. 
Units) 

Intervention Control 

Group Group 


Impact 


RGR 


Word Attack 


93.2 


0.45 


97.3 


94.7 


0.18 


0.36 


2.7 * 


0.50 


TOWRE PDE 


81.0 


1.27 


88.0 


86.3 


0.80 


0.91 


1.7 


0.13 


Word Identification 


88.7 


0.75 


91. 1 


91.7 


0.60 


0.56 


-0.6 


-0.07 


TOWRE SWE 


84.2 


1.05 


88.6 


87.2 


0.76 


0.85 


1.4 


O.II 


AIMSweb 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


NA 


Passage Comprehension 


92.6 


0.49 


91.0 


92.2 


0.60 


0.52 


-I. I 


-0.15 


GRADE 


91.2 


0.59 


88.4 


87.7 


0.77 


0.82 


0.7 


0.06 



Note: RGR is defined as RGR=(Impact/100-Average for Control Group at follow-up). 

Note: Gap is defined as (100-Average Score)/15, where 100 is the population average and 15 is the population standard deviation. 
Note: Values for AIMSweb are not available because normed standard scores are unavailable. 



Note: According to the first row of estimates, students in the third-grade cohort achieved an average standardized score of 92.4 on the Word 
Attack test at “baseline,” that is, shortly after the beginning of third grade — the intervention year. One year after the intervention year, that is, at 
“follow-up,” the students participating in any intervention achieved an average standardized score of 97.4, and the students in the control group 
achieved an average standardized score of 92.1, implying a statistically significant impact of 5.3 points. The “gap at baseline,” measured as the 
difference between the population average (100) and the study sample average (92.4) divided by the population standard deviation (15), was 0.5. 
One year after the intervention year, the gap was reduced 68 percent (see the “RGR”), when the reduction is measured as the impact (5.3) divided 
by the difference between the population average (100) and the control group average (92.1). The calculations described in this note might 
produce results that are slightly different from the estimates in the table due to rounding. 

* Impact is statistically significant at the 0.05 level. 
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B. Future Reports 



Reading Comprehension Interventions 

The decision to conduct an evaluation of the efficacy of reading comprehension interventions for 
informational materials in content areas such as social studies or science resulted from a series of 
discussions between the IRP and reading experts, as well as from the advice of a separate expert panel 
convened to identify important and policy-relevant evaluation questions to study in reading. The expert 
panel’s advice was that there are increasing cognitive demands on student knowledge in middle 
elementary grades where students become primarily engaged in reading to learn, rather than learning to 
read. Children from disadvantaged backgrounds lack general vocabulary as well as vocabulary related to 
academic concepts that enable them to comprehend what they are reading and acquire content 
knowledge. They also do not know how to use strategies to organize and acquire knowledge from 
informational text in content areas such as science and social studies.* The panel advised that strategies 
for improving comprehension are not as well developed as those for decoding and fluency. While there 
are multiple techniques for direct instruction of comprehension in narrative text that have been well- 
demonstrated in small studies, there is not as much evidence on teaching reading comprehension within 
content areas. 

This evaluation of reading comprehension is addressing the following questions: 

• Can promising reading comprehension interventions improve student reading achievement of 
informational text? 

• What are the most effective reading comprehension interventions for improving student reading 
achievement of informational text? 

• Under what conditions and practices do reading comprehension interventions improve student 
reading achievement of informational text? 

Five interventions were competitively selected by an expert panel and were piloted in three 5th grade 
classrooms each during the 2005-06 school year. Selection of the interventions was based on existing 
research evidence, quality of the intervention approach, capability to implement the intervention, and 
appropriateness of the intervention for the target population. All of the selected interventions 
supplement the core reading curriculum in teaching reading comprehension of text containing 
information such as science or social studies content. At the end of the 2005-06 school year, four 
interventions were selected to participate in the full-scale evaluation beginning in the 2006-07 school 
year. Those interventions and their publishers are: 

• CRISS (Project CRISS): CRISS teaches a wide array of comprehension and note-taking 
strategies using science text. Students then apply the strategies to the actual texts used in their 
classrooms. The program teaches students the difference between reading a text for basic 
information, reading for understanding a physical or natural phenomenon, and how to create 
succinct summaries. It also stresses active reading strategies such as asking oneself questions 



National Institute of Child Health and Human Development (2000). Report of the National Reading Panel, Teaching Children 
to Read: An Evidence-Based Assessment of the Scientific Research Eiterature on Reading and Its Implications for Reading Instruction 
(NIH Publication No. 00-4769). Washington, DC: US Government Printing Office. 
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while reading and then jotting down the answers. The program is designed to be used for 30 
minutes each day in the beginning of the year and then to be incorporated into aU content areas. 
Teachers participate in three days of initial training and one day of foUow-up training. 

• ReadAbout (Scholastic): Students are taught reading comprehension skiUs such as author’s 
purpose, main idea, cause and effect, compare and contrast, summarizing, and inferences 
primarily through a computer program. The program is designed to be used for 30 minutes per 
day, and students apply what they have learned during this time to a selection of science and 
social studies trade books. Teachers receive two days of initial training plus two additional days 
during the school year. 

• Read for Real (Chapman University; Zaner-Bloser): In Read for Real, teachers use a six- 
volume set of books to teach reading strategies appropriate for before, during and after reading 
such as previewing, activating prior knowledge, setting a purpose, main idea, graphic organizers, 
and text structures. Students use the materials for 30 to 45 minutes per day. Each of these units 
includes vocabulary, fluency, and writing activities. Teachers participate in three days of initial 
training and two, one-day foUow-up training sessions. 

• Reading for Knowledge (Success for AU Foundation): Reading for Knowledge, a 30-minute 
daily program, makes extensive use of cooperative learning strategies and a process caUed 
SQRRRL (Survey, Question, Read, Restate, Review, Learn). Teachers receive two days of initial 
training in addition to monthly follow up sessions. 

Eighty-nine schools within nine districts were randomly assigned to one of the four reading 
comprehension interventions or to a control group prior to the 2006-07 school year. Each intervention 
is being implemented in all 5* grade classrooms in each school. The impact of the interventions on a 
standardized reading comprehension assessment as well as informational texts in science and social 
studies wiU be estimated. The first report on the effectiveness of the reading comprehension 
interventions is expected in Spring 2008. 

Mathematics Curricula 

The decision to conduct an evaluation of the effectiveness of mathematics curricula resulted from a 
series of discussions with and recommendations from the IRP, the Office of Elementary and Secondary 
Education, and an expert panel convened to provide advice on poUcy-relevant questions that would be 
important to address in an impact evaluation focused on mathematics. Information on the effectiveness 
of mathematics curricula is crucial to improving performance on state mathematics assessments under 
NCLB. There is considerable controversy about what mathematics children should learn and how it 
should be taught, but there is very Utde reliable information available to educators and policy makers 
about which curricula are most likely to improve mathematics achievement.* 

This evaluation is focusing on early elementary grades because disadvantaged children are behind their 
more advantaged peers in basic mathematics competencies even before entering elementary school. If 
basic concepts are not mastered in early elementary grades, students have great difficulty understanding 
more advanced mathematics concepts in upper elementary grades. The evaluation wiU compare different 



The National Academy of Sciences (2004). On Evaluating Curricular Effectiveness: ]udging the Quality ofK-12 Mathematics 
Evaluations. Washington, DC: The National Academies Press. 
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approaches to teaching early elementary mathematics since there are many mathematics curricula that are 
being widely implemented without evidence of their effectiveness. 

The major evaluation questions are: 

• What is the relative effectiveness of different math curricula on student achievement for early 
elementary school students in disadvantaged schools? 

• Which math curricula result in sustained impact on student math achievement? 

These questions are being addressed through an experimental methodology in which schools are randomly 
assigned to selected math curricula. The study design calls for recruitment of 1 0-1 5 school districts with a 
total of 100 schools across all sampled districts. Within each sampled district, each sampled school has been 
randomly assigned to one of the four curricula. Since all schools have a core mathematics curriculum, 
including the curricula being tested in the study, there will not be a no-treatment control group. For some of 
the schools, the intervention curriculum is their regular curriculum. The difference in math achievement 
among the schools using each math curricula will be the measure of the relative effectiveness of each math 
curriculum on student achievement. 

• What is the relationship between teacher knowledge of math content and pedagogy and the 
effectiveness of math curricula? 

Teacher knowledge of mathematics content and pedagogy is being assessed prior to training on the 
curriculum that their school is implementing allowing for a subgroup analysis of the impact of the 
curricula on student achievement taught by teachers with varying expertise. It is hypothesi 2 ed that some 
math curricula may require higher levels of teacher knowledge than others for successful 
implementation. The teacher assessment being used is “Learning Mathematics for Teaching” developed 
by the Consortium for Policy Research in Education for the Study of Instructional Improvement. 

• Under what conditions is each math curriculum most effective? 

This question will be addressed by correlational analyses. Fidelity of implementation of each curriculum 
win be measured, as well as characteristics of students, teachers, schools, and districts. These measures 
win be correlated with the relative impacts of the math curricula on student achievement. 

Four commercially-available mathematics curricula were competitively selected by an expert panel to 
represent a variety of approaches to teaching mathematics. Selection of the interventions was based on 
existing evaluation evidence, extent of use of curricula in schools, capabiUty to implement the 
intervention, and appropriateness of the intervention for the target population. These curricula 
represent varying approaches to mathematics instruction and include widely used curricula. The 
curricula are: 

• Investigations in Number, Data, and Space (Pearson Scott Foresman) is a child-centered 
approach to teaching mathematics through activities, discussions, and problem solving. Students 
are involved in meaningful mathematical problems, and teachers engage in ongoing learning 
about mathematics content and how children learn mathematics. 
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• Math Expressions (Houghton Mifflin) combines conceptual understanding with opportunities 
to develop fluency with problem solving and computation. Both reform and traditional 
mathematics approaches are incorporated along with new teaching strategies. 

• Saxon Math (Harcourt Achieve) provides a multi-sensory approach designed to enable all 
children to develop a solid foundation in the language and basic concepts of mathematics. This 
is accomplished through hands-on activities that actively engage students. Concepts are 
reviewed and practiced over time leading to mastery and fluency. 

• Scott Foresman-Addison Wesley Mathematics (Pearson Scott Foresman) provides explicit 
instruction of essential mathematics skills and concepts, using concrete manipulatives and 
pictorial and abstract representations. Ongoing assessment and diagnosis are coupled with 
strategic intervention to meet the needs of individual students. 

The evaluation began in the 2006-07 school year with curricula being implemented in the first grade. 

The curricula will be implemented in the first and second grades during the 2007-08 school year and 
extended to the third grade in the 2008-09 school year. 

Four school districts with forty schools have agreed to implement the curricula in all first grade 
classrooms during the 2006-07 school year. Data wiU be collected on student mathematics achievement 
at the beginning and end of the 2006-07 school year and on implementation of each curriculum in the 
spring of 2007. The student mathematics assessment developed for the Early Childhood Longitudinal 
Study wiU be used in this study. This assessment is individually-administered and computer adaptive. 

Additional districts with approximately 60 schools are being recruited to begin participation in the study 
in the 2007-08 school year. The additional schools wiU implement the curricula in their first and second 
grades. The original 40 schools wiU continue implementation of the curricula in their first grade 
classrooms as well as in their second grade classrooms. Data wiU be collected on the implementation of 
each curriculum and student mathematics achievement for first- and second-graders during the 2007-08 
school year. The first report on the relative effectiveness of the mathematics curricula is planned for 
Spring 2009. 
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Appendix A 

National Assessment of Title I 
Independent Review Panel Members 



Kaleem Caire, Next Generation Education Foundation 

Tom Cook, Northwestern University 

Chris Cross, Cross &Joftus, LLC 

Gayle FaUon, Houston Federation of Teachers 

David Francis, University of Houston 

Eric Hanushek, Hoover Institution, Stanford University 

Sharon Johnson, Withrow University High School 

Paul Peterson, Harvard University 

Eric Smith, College Board 

John Stevens, Texas Business and Education Coalition 
Patricia Supple, Archdiocese of Los Angeles 
Tasha Tillman, Black Alliance for Education Options 
Maris Vinovskis, University of Michigan 
Rodney Watson, Louisiana Department of Education 
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